Evaluation of Techniques for Structuring Multi-Criteria Decision Problems

This article seeks to overcome the problem of structuring multi-criteria decision problems in a scientifically valid way. For this purpose, we theoretically and empirically compare two techniques which can be used for the purpose of structuring problem domains: card sorting procedures and statistical web mining. Based on two empirical test cases we assess whether decision structuring is reliable regarding the applied structuring method and whether the resulting hierarchies are valid representations of the decision problem at hand. The results indicate that the two techniques lead to quite different goal-criteria hierarchies and that web mining does not produce useful problem representations. In contrast, card sorting seems to be a valid structuring technique. We explain these results by the fact that card sorting procedures are interpretive techniques which are able to deal with vague concepts (criteria) while web mining, as a purely statistical approach, does not work well with ambiguous concepts.